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ABSTRACT 

This paper describes a Web-based system to support 
temporary Anomaly Response Teams formed from 
distributed subteams in Space Shuttle and International 
Space Station mission control, to handle anomalies during 
missons. The system was designed for easy and flexible 
creation of small collections of files and links associated 
with work on a particular anomaly. The system supports 
privacy and levels of formality for the subteams. First we 
describe the supported groups and an anomaly response 
scenario. Then we describe the support system prototype, 
Anomaly Response Tracking and Integration System 
(ARTIS). Finally, we describe our evaluation approach, and 
results of the evaluation. 

Keywords 
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World Wide Web, groupware. 

INTRODUCTION 

Within organizations, a number of types of computer-based 
library services can be provided t> support task-oriented 
teams [4, 5, 7, 9, 10]. We were asked to design a system to 
support temporary Anomaly Response Teams formed from 
distributed subteams in mission control, to handle 
unexpected anomalies when they occur during Space 
Shuttle or International Space Station missions. Although 
the teams are temporary, the analyses and decisions 
produced by each team would be additions to an 
organizational memory, an anomaly response information 
archive. Our management sponsor wanted an archive of 
analyses done for and by anomaly response teams, so that 
relevant analyses from past work on anomalies could be 
found and applied to current cases. 

Our design solution was a World-Wide-Web-based digital 
library and team memory. We used an iterative design 
approach, preceded by a study of Anomaly Response Team 


Klaus Christoffersen, Renee Chow 

Cognitive Systems Engineering Laboratory 
The Ohio State University 
1971 Neil Ave., Columbus, OH 43210 

activities, based on observations, interviews and critical 
incident analysis of past cases [13]. In this work we 
encountered many of the issues that have been addressed 
in the work on organizational memories and digital libraries. 

Because anomaly response teams work in fast-paced time- 
constrained situations, a main driver for the design was ease 
of use and minimization of work in creating and managing 
the collections of heterogeneous information that are used 
and produced by Anomaly Response Teams. As noted by 
Grudin [8], Rein et al. [10], Braa et al. [5] and Sorgaard [1 1], a 
supportive computer service requires work from its users. 
The cost of the work must not be too high for the benefit. 

The system can fail because it is not worth enough to any 
one individual to reach a critical mass of usage [8]. In other 
words, the anomaly response information archive would be 
fragmentary if the support tool was not continuously useful 
and easily usable by team members during the mission. As 
Gronbaek et al. [7] note, users do not tend to see a 
cooperative work support system as anything other than 
support for their individual work. The system should feel 
like an individual workspace that also provides access to 
closely related work by other team members. The delimited 
and focused nature of Anomaly Response Teams helps 
keep collections small and relevant. 

The emphasis on ease of use made it necessary to simplify 
or eliminate a number of functions of digital or Web-based 
libraries. Because of the heterogeneous information and 
processes used and produced by the subteams, a simple 
flexible approach was needed (cf. Grudin’s exception 
handling challenge, 8). Users would not create documents or 
files in a standard format, and would rarely be collaborative 
authors. Indeed, the collection could provide a workspace 
for collecting both links and files in several formats. Thus, 
requirements for version control and locks, authoring 
standards and work flow process management could be 
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minimized or eliminated [10]. Users would not be required to 
create structure in the collection, a difficult task [9]. 
Collection maintenance [3, 10] could be minimized because 
these collections are historical rather than generic, and 
would include contextual information provided by 
contributors of files or links [1,9]. 

The nature of the distributed Anomaly Response subteams 
meant that the design would accommodate informality and 
privacy [I, 8]. Mechanisms for incremental entry and 
promotion of elements of the collections were used to 
support levels of formality and privacy. They also provided 
a simple form of collection maintenance as a by-product. 

The problem of adoption [5, 8, 11] was addressed by our 
iterative design process and by integrating the support 
system with a new simple capability to capture interpreted 
mission data from control center workstation applications. 

This paper describes the support system prototype, 
Anomaly Response Tracking and Integration System 
(ARTTS). First we describe the supported groups and an 
anomaly response scenario. Then we describe the support 
provided by the prototype. Finally, we describe our 
evaluation approach, and the results of the evaluation. 

COOPERATIVE ITERATIVE DESIGN APPROACH 

In our design work on intelligent systems software, we use 
an iterative method for analysis, design, prototyping and 
evaluation [12]. Since the majority of the method is generally 
applicable to innovative advanced software, we used this 
method in development of ARTIS. The method is oriented to 
incremental development and use of task descriptions and 
scenarios, prototypes, evaluations and requirements. 
Throughout this process, the understanding of the 
activities, environment and problems of the users is refined. 
Early work focuses on getting he goals and objectives 
right, to guide design and evaluation of a useful software 
product, but refinement of this understanding continues 
throughout the project [14]. This understanding is embodied 
in descriptions and scenarios, rather than formal models. 
Informal evaluations of prototypes by user representatives 
occur throughout. Finally, a formal evaluation is conducted 
when the prototype is mature enough to be likely to be 
judged useful and usable by the evaluators. After this 
evaluation, a brief requirements document is written that 
refers to the prototype and incorporates any changes that 
are recommended as a result of the evaluation. 

Our project team included representatives of the two primary 7 
user groups, a former Mission Control Center (MCC) flight 
controller and a former Mission Evaluation Room (MER) 
engineer. The flight controller represented the management 
sponsor and the interests of flight controller users 
throughout the project. She provided the majority of the 


informal evaluations. The engineer was responsible for 
detailed design and implementation of the final prototype. 

Our early analysis took the form of a study of anomaly 
response teams. This provided the initial focus on support 
for team coordination with a central repository of working 
information. Informal evaluations of early prototypes helped 
us learn that privacy and formality issues were important to 
the design, because the two primary user groups were 
accustomed to working separately until their positions were 
mature enough to present to the other group. As users saw 
the utility of the repository, they turned to consider ease of 
use, and made clear that is was a make-or-break issue in the 
fast-paced world of responding to space mission anomalies. 
Our concept matured to include a set of easy steps, varying 
in effort to correspond to the likely pace of activity during 
the anomaly response process. Late in design, the users 
sought integration and consistency with activities on 
console in the control center. The concept of simple window 7 
shapshots to capture interpreted mission data was the 
result. 

ANOMALY RESPONSE TEAMS - THE SUPPORTED 
GROUPS AND THEIR DIFFICULTIES 

This section begins with a summary of the design-relevant 
findings of the Watts et al. [13] study of Space Shuttle 
anomaly response activities. This study was based on 
observations, interviews and critical incident analysis [6] of 
past cases. The flight controllers and supporting engineers 
that make up the principal subteams in anomaly response 
begin their work independently, since each group is located 
in an independent ground support control room during the 
mission. When a problem is identified as an unexpected 
anomaly without a corresponding set of standard operating 
procedures, the system specialists from the two subteams 
begin working directly together. In coordinative meetings, 
each subteam brings information, analyses and 
recommendations on what should be done. Because the 
flight controllers on console in the control center cannot 
leave to attend meetings, a new team of flight controllers 
(“team 4”) is assigned to handle the coordination and team 
meetings. The diverse roles, resources and information from 
each group are helpful to the process of anomaly response. 
Argumentation and advocacy concerning conflicting 
interpretations and positions help make the decision 
process thorough and broad. Additional team members 
outside of these two main groups come and go, to provide 
specialized information or to oversee the process from the 
perspective of the whole mission. People responding to the 
anomaly can be geographically and temporally dispersed. 

The primary support strategy is to make it possible for 
people involved in the anomaly response process to share 
information with one another. There is need for an organized 
central repository of information and data, old or new. 
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located or created. A large amount of information can be 
located, produced, tracked, distributed and used by various 
individuals for various purposes in the time-constrained 
anomaly response situation. Some transfer of documents 
and information is accomplished by walking around. Some 
paper-based and electronic archival activities occur during 
and after the mission. The information is not managed in a 
coordinated way across subteams. It may be hard to follow 
all the relevant activities because they are numerous and 
distributed. Members with urgent needs for new information 
may not know in a timely manner whether and where it is 
available, especially if they miss a meeting. 

A digital means of collecting and managing hformation 
could provide significant benefits to anomaly response 
teams. The primary challenge is to provide that support in a 
way that makes it easy to use when the tempo of activity is 
high during the mission, and that produces an archive where 
it is easy to locate relevant information from past anomaly 
response activities. 

A TASK SCENARIO AND ROLE OF ARTIS 
This generic task scenario describes the situation from the 
point of view of flight controllers. A flight controller on a 
discipline console (specializing in a defined set of space 
systems) in the MCC front or back room observes a 
possibly anomalous event or trend in the real time data. The 
flight controller communicates this information via phone 
and voice loops to other flight controllers and the Flight 
Director as needed. (The Flight Director is the primary 
person in charge of mission control activities to support a 
flight.) 

As time permits, the flight controllers manning the discipline 
consoles use various on-console tools such as plotters, 
data logs or playback to analyze data relevant to the 
anomaly. The flight controller begins work on a document to 
provide status information on the problem or anomaly. The 
flight control discipline gets an action from the Flight 
Director to develop a position on the anomaly. 

The flight controller sends a “CHIT” request to the 
supporting engineers in the MER, a parallel engineering 
center, requesting assistance in analyzing aspects of the 
anomaly or its impacts. The engineers begin work on 
developing positions fora CHIT response. 

An anomaly response team is formed, including “team4” 
flight controllers from the discipline, MER engineers and 
others from outside who can assist. Actions are assigned to 
gather all relevant information from the current mission and 
previous missions, and to perform analyses. Team members 
and others in the discipline on and off site develop the 
needed products. They work to isolate the problem, plan 
responses, run tests on the ground or in flight, develop new 


procedures, change flight rules, and so on, depending on 
the nature of the anomaly. Team 4 members report to the 
flight controllers on console in the MCC. Anomaly response 
team meetings coordinate the work during the mission until 
issues relating to the anomaly are resolved. 

There can be several independent anomalies during a 
mission, and a parallel set of Anomaly Response Teams can 
be at work. Each team reports its progress daily to the 
Mission Management Team (MMT), and this is recorded in 
the MMT minutes. 

Flight controllers will provide the ARTIS database with the 
initial description of an anomaly by capturing the display of 
mission data indicating that an anomaly exists. In addition, 
they will be consumers of ARTIS information by tracking 
what analyses have been done, what actions are 
recommended, and whether meetings are scheduled for 
investigating the anomalies. 

Other “team 4” flight controllers support the mission but are 
not assigned to one of the three shifts which cover console 
operations during a 24 hour period. Until these flight 
controllers have been assigned a specific anomaly -related 
task, they will be simply consumers of ARTIS information, 
checking in periodically to maintain mission cognizance and 
to see what anomalies are being worked and what types of 
analyses are being performed. Upon receiving an anomaly 
response task, they will begin using ARTIS more actively, 
looking for information related to the anomaly and 
collaborating with others in performing and publishing 
analyses to support the anomaly response process. The 
team 4 flight controllers currently need to call in to the 
console, visit the console, and examine written console logs 
to keep abreast of mission events. With ARTIS, they will be 
able to reduce the number of visits to the console, thereby 
improving their own productivity as well as that of the flight 
controllers actively working on console. 

MER engineers will have a similar role to the team 4 flight 
controllers, with respect to using ARTIS. Like the team 4 
flight controllers, they will be primarily consumers of ARTIS 
information until they receive a specific request for an 
analysis. Remote off-site engineers who are called into the 
anomaly response process can also use ARTIS remotely 
because it is Web-based. 

PROTOTYPE DESIGNS - SUPPORT FOR TASKS 

The ARTIS prototype is implemented using a browser 
interface to a database. The prototype is operational, but 
only one Workspace is populated, with example data from 
one anomaly case. 

ARTIS assists anomaly response team members in 
performing the following tasks: 
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• Find current anomaly information 

• Find archived information related to the current 
anomaly 

• Share anomaly information and analyses with other 
anomaly response team members 

• Manage ARTIS collections 

Find Current Anomaly Information 


Finding current anomaly information is one of the primary 
motivations for anomaly response team members to use 
ARTIS. A team 4 flight controller can examine mission data 
describing an anomaly and search the ARTIS archives from 
the office. A MER engineer can check the list of action items 
and meeting notices to monitor other anomaly response 
activities. 



Figure 1. The Initial View Shows The Most Commonly Used Information And Links. 


What anomalies are being watched? 

The first display the user sees after logging into ARTIS is 
shown in Figure 1 . Across the top of the display are links to 
“Current Mission”, “Past Missions”, “Search”, 

“Administration”, “MIS” (an external information source), 
and “SSVEO” (another external information source). 

Because most of the interactions with ARTIS will involve 
the current mission, the current mission area is the default 
first screen after log-in. The “Workspaces” listed below 
indicate anomalies which are being worked for the current 
mission. Thus, upon logging into ARTIS the user 
immediately' sees what anomalies are being worked for the 
current mission, key information for maintaining mission 
cognizance. Clicking on the “info” link will reveal a short 
description of the anomaly. Clicking on the name of the 
anomaly will access the Workspace (collection), and show 
all the documents, files, and links that have been entered. 


The column identified as “Owner Group” refers to a specific 
flight control or engineering discipline, which also 
corresponds to group membership for the purposes of 
sharing private documents (described later). 

What Has Been Done To Investigate The Anomalies? 

The display in Figure 2 results from clicking on the name of 
a specific anomaly, “APU 3 High Fuel Pump Inlet Pressure”, 
in this case. By showing the names of all the documents, 
files, and links entered under that anomaly, it answers the 
question, “What has been done to investigate the 
anomaly?” 

Based on the log-in information, ARTTS identifies the group 
to which the individual belongs, thereby determining which 
“Private” files to show. The private files are visible to only 
group members so that drafts can be generated without 
exposing them to the entire anomaly response team. This 
allows collaborators to share ideas freely to generate 
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analyses and reports without risking misunderstandings documents in the Private area of ARTIS, collaborators need 
because they are still in draft form. By entering these draft 


Global documents are 
visible to every member of 
the anomaly response team. * 


Private documents allow local 
groups to collaborate on draft 
documents before they are ready 
to be seen by the rest of the team. 


“Workspace 
Documents” 
menu to select 
subsets of the 
list shown at the 
right. 

“Document 
Management” 
menu to add 
documents and 
links to the 
workspace. 

“Action Item” 
menu to view 
and edit the 
action item list. 



Figure 2. The Major Display Regions and Menus of ARTIS 


not be physically co-located. For instance, a flight 
controller on console and two others in their offices could 
collaborate on a single report, using ARTIS to update the 
draft and to share notes and comments. 

What were the outcomes (meeting notes, reports, action 
items) of previous meetings on this anomaly? 

In Figure 2, the “Workspace Documents” selection called 
“All Workspace Documents” has been highlighted. This 
means that all categories of documents for this anomaly are 
being displayed at the right. To see a subset of the related 
documents, the user clicks on one of the categories in the 
Workspace Documents menu. For instance, clicking on 
“Meeting Notices, Summaries and Status” would cause 
only the line “MMT Minutes - 3/6/94” to be displayed at 
the right. This helps the user to eliminate clutter and isolate 
the objects related to specific information needs. 

When is the next meeting to discuss an anomaly? 

In this case, the user is interested to know if there is an 
upcoming meeting to attend. By clicking on “Meeting 
Notices, Summaries and Status” and seeing only one 
meeting report, she can quickly and confidently conclude 
that no meetings have been scheduled yet. 


Find Related Archived Information 

Finding related information was the specific assistance 
requested by the manager at the beginning of the ARTIS 
project. When an anomaly is encountered, access is 
needed to previous analysis reports that are relevant to the 
current anomaly. Some analyses consist of involved 
computer simulations; others involved physical 
experiments to test the operational characteristics of 
Shuttle hardware under very specific constraints. If a 
similar analysis has been performed in the past, much time 
can be saved in the anomaly response process. The user 
navigates to the related information by using the links that 
appear at the top of every ARTIS display (Past Missions, 
Search, MIS, and SSVEO). 

By selecting the “Past Missions” link, the user can browse 
ARTIS databases for previous missions. By selecting the 
“Search” link, the user can search the archived ARTIS data 
for previous missions. The following search indices can be 
automatically associated with collections to support 
keyword search: vehicle ID, flight number, discipline 
(EECOM, MMACS, DPS, etc.), crew, and type of mission 
(e.g., spacelab, MIR docking, space station, type of 
payload). 
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ARTIS links to two other existing databases. The first is 
called Mission Information System (MIS), which provides a 
short synopsis of many anomalies and recommendations 
from previous missions. The second is called the Space 
Shuttle Vehicle Engineering Office (SSVEO) database, and 
it contains general Shuttle reference material plus 
information from the engineering office about the current 
mission. 

Share Anomaly Information and Analyses with Other 
Anomaly Response Team Members 

The initial request was for access to archived information, 
especially analysis reports, from previous missions. We 
faced a classic database problem, how to populate it with 
quality data. Our strategy was to serve current mission 
needs with the same actions that build an archive of the 


current mission. In the previous section, we showed how a 
web-based repository of information addressed the needs 
of finding current and past information. In this section, we 
show the strategy for making it easy to enter information 
about the current missions into ARTIS, without violating 
privacy concerns, and without requiring much effort to 
convert the current mission data into a searchable archive 
that can serve future missions. 

Save mission data from the console 

First, it is important for the flight controllers on console to 
be able to show the nature of the anomaly to the remainder 
of the anomaly response team. Figure 3 represents a 
hypothetical display of a portion of the console screen with 
mission data. 


Flight 

controllers ’ main 
display 
navigator 
(DNAV) (also 
Non-ARTIS). 


Non-ARTIS 
plotting software 
display. 

File naming 
window. 



Figure 3. Window Snapshots of Mission Data can be Uploaded to ARTIS 


The window titled MIS Plot contains an annotated data plot 
which the flight controller constructs using existing (non- 
ARTIS) software. To save this for ARTIS, the flight 
controller selects the Window Capture option from his/her 
main menu (the window marked DNAV). This selection 
causes the window to the right of the display to appear, 
showing a default filename (containing a date-time stamp for 
identification). If the controller b not rushed for time, she 
can rename the file to something more meaningful (e.g,, 
APU_post_insertion.gif). Because the flight controller can 
be very pressed for time during this task, it was important to 
make the capture as quick as possible, hence the default 
filename, which the controller can OK without much 


thought. Once the filename has been entered, the cursor 
changes to a target and the controller clicks on the window 
to be captured. 

Collection areas for these files are provided in the console 
environment, so that a large number of potentially useful 
window captures can be taken before actually deciding to 
enter selected flies into ARTIS collections. Once an ARITS 
Workspace is created, relevant files can be uploaded into 
ARTIS for viewing by the remainder of the anomaly 
response team. 
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Unload files and links to the ARTIS database 
Once a team member has produced a file containing an 
image or document to be shared with the remainder of the 
team, it can be uploaded to the ARTIS database. The 
display in Figure 4 resulted from clicking on the “Add New 
Documents” option under the “Document Management” 
menu at the left side of the display. In the interest of making 
the form as easy as possible to complete, default entries 
were already made, based on log-in and navigation 
information for all but the file name, descriptive name, and 
comments. All other entries are editable. For entries like 
“category” which have a limited set of options, those 
options were placed in a drop-down list to make them easily 


selectable. The document being entered in Figure 4 was 
entered as a “Private” document. 

Some of the work of the team involves locating relevant 
information that is already available on the Web. A similar 
form is used to add links to other web pages by selecting 
the “Add New Links” option from the Document 
Management menu at the left. The user provides a 
descriptive title and may provide comments on the Web link. 
These forms provide some context and meta-data for each 
file and link. Even though externally linked information may 
change, the context information can still be useful. 
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Figure 4. Files can be Uploaded to the ARTIS Database 


Collaborate with group members 

Collaboration with group members occasionally requires 
some privacy, while making the document available to 
people at multiple locations. Flight controllers on console 
and in offices may want to make drafts available to one 
another without showing them to the remainder of the 
anomaly response team. This is the purpose of a private 
workspace. When a person logs into ARTIS the association 
of that person with a private group is established 
automatically (e.g., user “tbery” has been associated with 
the MMACS flight controller group, so the only private 
documents he can see are MMACS flight controller private 
documents). 


Promote analyses and documents to team-wide workspaces 

Once a draft becomes final, it is time to move the document 
from the private workspace to the global workspace so the 
whole anomaly response team can see it. Figure 5 shows 
how a document currently restricted to private viewing by 
collaborators can be promoted to be globally accessible 
when ready. For instance, if the “MOD Rationale for 
Minimizing APU 3 Run” is now ready for viewing by the 
entire anomaly response team, the author can click on the 
“Move” button under the Promote column of the Private 
table below. Afterwards, any member of the anomaly 
response team can view it. The “Demote” option was 
provided in the Global area so documents can also be 
moved back to the Private area. 
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Figure 5. Private Documents can be Promoted for Global Access and Archiving when Ready. 


Manage ARTIS Collections 

Managing the ARTIS collections is a task that the anomaly 
response team does not currently have, so it is important 
that the management responsibilities are not burdensome. 
The nearest manual correlate they have is binding the 
analyses, reports, meeting notes, etc., into a 3-ring mission 
binder for future reference. Fortunately, there are not many 
new management tasks introduced by ARTIS. Adding users 
and assigning them to groups is a task that shoulu not be 
performed often because the membership of each group 
tends to be stable. For instance, turnover in the MMACS 


flight controller group is not so high that maintaining a list 
of members of that group is burdensome. To add new users 
or groups, the administrator selects the appropriate menu 
option under the Administration menu (shown in Figure 6). 

Figure 6 illustrates how a new workspace is created. This is 
a task that can be performed under some time pressure, so it 
needs to be streamlined. ARTIS fills in as much default data 
as possible, leaving the user to enter a workspace name and 
a workspace description. If the user needs to edit the 
default values, she simply selects that entry area and types 
over it. 


Create New Workspace . 



Figure 6. A Workspace for a New Anomaly can be Created with Minimal Effort. 


At the end of a mission, global workspaces are kept as 
archives that can be referenced during future missions. The 


data collection areas and private areas of workspaces are 
purged. 
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EVALUATION METHOD AND RESULTS 
Our approach to formal evaluation of prototypes is designed 
for advanced software for novel uses such as the multi-user 
applications for computer-supported cooperative work [12]. 
Because failures of such software are often due to 
misunderstanding of multi-user tasks and their difficulties, 
and problems of adoption [8], our evaluation focused on 
refining our understanding of the activity and getting some 
indicators of the utility or usefulness of the support strategy 
used in system [14]. 

At the usability level, the primary concern is with questions 
about implementation of the chosen aiding strategy (e.g., 
"should the database use queries or hypertext links?" "what 
should the structure of the menus be?"). While we were 
careful to develop a design with good usability, we chose to 
focus the evaluation on our understanding and the 
usefulness of the system. We noted that the strategies that 
we developed to achieve ease of use were above the level of 
traditional usability concerns. In our system, ease of use 
was a utility or usefulness issue. 

Six current or former space shuttle flight controllers served 
as participants in the evaluation. The core of the evaluation 
consisted of demonstration-style walkthroughs of several 
anomaly response scenarios intended to showcase the main 
functions of ARTIS. The intent was thus to test the 
relevance of our design basis scenarios as well as to obtain 
feedback about how ARTIS addressed these scenarios. 
Participants were encouraged to "think aloud" as much as 
possible and to offer feedback on any and all aspects of the 
prototype. All of the sessions were audiotaped and 
subsequently transcribed. 

Each session began with a "critical incident review" [6], in 
which participants were asked to recount the events of the 
last serious anomaly they had worked on ("serious" was 
defined as an anomaly where the team 4 group had been 
called in). They were asked to focus on the nature of how 
information had flowed among members of the anomaly 
response team and what, if any, difficulties were 
encountered. 

Next, we solicited feedback from the participants on a 
schematic flow diagram depicting our conception of the 
major components of the anomaly response process. This 
diagram also included a list of what we thought of as the 
most important information resources supporting the 
anomaly response process as well as a list of the most 
significant bottlenecks in the process. Overall, the results of 
these exercises served to substantially reinforce our 
confidence in the model of anomaly response activities on 
which ARTIS is based. 


The participants generally reacted positively to the idea of a 
centralized, remotely accessible repository of information 
about current and past anomaly response activities. There 
was unanimous agreement that ARTIS could represent a 
considerable improvement over some of the current methods 
of information capture and distribution during anomalies. 

In the final portion of the evaluation, participants were given 
a chance to describe, in an open-ended fashion, how they 
thought ARTIS might or might not be useful. The following 
excerpt is representative of the majority of the responses: "It 
[ARTIS] certainly makes all the information from current 
flight, previous flights available in one common location and 
it allows the flexibility of anyone adding to this at any time. 
We don't have anything right now that does this. ... At least 
during the [STS-] 76 anomaly there was so much going on, 
but there was no common place to get the information. 
That's... what slows things down... and you can't really 
follow what’s going on unless it's available... [it] takes a long 
time to figure out where all the pieces are coming from." 
Thus, in general, the evaluation served to substantiate and 
refine our understanding of the anomaly response process, 
as well as to reinforce our confidence in the usefulness of 
the assistance that ARTIS provides for anomaly response 
activities. 

The evaluation uncovered concerns about how the 
database would be populated initially with information from 
past anomalies, and how it would be maintained. With 
clerical assistance, paper documents currently kept in 3-ring 
binders could be converted to ARTIS format to make them 
more accessible and digitally searchable. 

There were also concerns that ARTIS might not be utilized if 
it forced a duplication of archival duties currently performed 
elsewhere. Thus, further work is needed on integration of 
ARTIS with currently used tools and work practices. 

CONCLUSIONS 

In this work, we found that a group support system can be 
designed for ease of use in a way that is distinct from 
usability. We found that to achieve an adoptable support 
system, we had to spend a significant amount of our effort 
on making a simple and flexible system. The work required to 
enter files and links and structure a collection has to be 
minimal. Although we considered providing a structured 
work process and tools to check consistency of hypotheses 
and evidence, we abandoned these concepts in favor of 
simplicity and flexibility. 

ARTIS provides a hybrid between an organizational 
information repository and a World Wide Web digital 
library. It contains document and graphics flies as well as 
links. Its contents could be authored by anyone, but 
members of the anomaly response teams control the 
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collections, Context information is associated with each file 
and link. The maintenance of the collections as Workspaces 
has to be minimal and almost invisible to the users. 
Versioning is handled simply, with promotion to the Global 
area signifying permanence in the archives. Since the 
archives are historical, there is not a continuing need to 
maintain them for consistency with current organizational 
knowledge. 

Currently, ARTIS is being re -implemented and extended to 
support new requirements of flight controllers for issue 
tracking. 
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